Papers with language identification
A reproduction of Apple’s bi-directional LSTM models for language identification in short strings (2021.eacl-srw)
Copied to clipboard
| Challenge: | Language Identification is the task of identifying a document’s language. |
| Approach: | They propose to use bi-LSTMs to identify language on very short strings such as text message fragments to perform automatic spell check. |
| Outcome: | The proposed model outperforms open-source language identifiers and its language identification mistakes are due to confusion between related languages. |
Data Filtering using Cross-Lingual Word Embeddings (2021.naacl-main)
Copied to clipboard
| Challenge: | varying task definitions and data conditions make it difficult to draw a meaningful comparison. |
| Approach: | They propose to use language identification to perform data filtering on MT data based on cross-lingual word embeddings to identify weaknesses in language identification tool. |
| Outcome: | The proposed methods perform well on three real-life, high resource MT tasks while performing weakly within more realistic task conditions. |
Measuring Linguistic Competence of LLMs on Indigenous Languages of the Americas (2026.eacl-short)
Copied to clipboard
| Challenge: | Existing benchmarks for linguistic knowledge of Indigenous languages of the Americas focus on high- and medium-resource languages with substantial digital presence. |
| Approach: | They propose a framework for probing large language models’ linguistic knowledge of Indigenous languages of the Americas using zero-shot prompting and few-shot probing. |
| Outcome: | The proposed framework evaluates models from five major families on 13 Indigenous languages including Bribri, Guarani, and Nahuatl. |
MaskLID: Code-Switching Language Identification through Iterative Masking (2024.acl-short)
Copied to clipboard
| Challenge: | Sentence-level LIDs are classifiers trained on monolingual texts to provide single labels, typically using a softmax layer to turn scores into probabilities. |
| Approach: | They propose a simple yet effective code-switching language identification method that uses the LID itself to mask features associated with L1 and L2 in the next round. |
| Outcome: | The proposed method is based on two open-source LIDs based in the FastText architecture and does not require any external resources. |
Normalization of Indonesian-English Code-Mixed Twitter Data (D19-55)
Copied to clipboard
| Challenge: | Twitter is an excellent source of textual data for NLP researches, but it is noisy and often contains typos, slang terms, and non-standard abbreviations. |
| Approach: | They propose a standardization system for Indonesian-English code-mixed Twitter data that includes tokenization, language identification, lexical normalization, and translation. |
| Outcome: | The proposed standardization system is based on four modules for tokenization, language identification, lexical normalization, and translation. |
Improving Pretraining Techniques for Code-Switched NLP (2023.acl-long)
Copied to clipboard
| Challenge: | Multilingual pretraining models for code-switched inputs are a key component of NLP applications. |
| Approach: | They propose to use masked language modeling techniques to mask code-switched text that are cognizant of language boundaries prior to masking. |
| Outcome: | The proposed techniques improve performance on two downstream tasks, Question Answering (QA) and Sentiment Analysis (SA), compared to standard pretraining techniques. |
Bhasa-Abhijnaanam: Native-script and romanized Language Identification for 22 Indic languages (2023.acl-short)
Copied to clipboard
| Challenge: | Existing tools for language identification are noisy, small and similar to high-resource languages. |
| Approach: | They create a language identification test set for native-script and romanized text which spans all 22 Indic languages and train a model for romanized script. |
| Outcome: | The proposed model improves on native-script and romanized script, and is competitive or better than existing LIDs. |
NLP Privacy Risk Identification in Social Media (NLP-PRISM): A Survey (2026.findings-eacl)
Copied to clipboard
| Challenge: | Social media platforms such as X (formerly Twitter), Facebook, and Reddit generate user-generated content. |
| Approach: | They propose a framework to assess privacy risks in social media by evaluating vulnerabilities across six dimensions: data collection, preprocessing, visibility, fairness, computational risk, and regulatory compliance. |
| Outcome: | The proposed framework assesses privacy risks across six dimensions . it achieves F1-scores of 0.58–0.84, but incurs 1% - 23% drop under fine-tuning . |
Universal Dependency Parsing for Hindi-English Code-Switching (N18-1)
Copied to clipboard
| Challenge: | Code-switching data often need additional processes such as language identification, normalization and/or back-transliteration to be processed. |
| Approach: | They propose a neural stacking model that leverages part-of-speech tags and syntactic tree annotations in tweets to parse code-switching data. |
| Outcome: | The proposed model is 1.5% better than the augmented model and 3.8% better than one which uses first-best normalization and/or back-transliteration. |
Improving Informally Romanized Language Identification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Latin script is often used to informally write languages with non-Latin native scripts, resulting in high spelling variability. |
| Approach: | They propose to improve methods used to synthesize training sets to incorporate natural spelling variations into training sets. |
| Outcome: | The proposed method improves test F1 from the reported 74.7% (using a pretrained neural model) to 85.4% (using the linear classifier trained solely on synthetic data). |
On the Language Neutrality of Pre-trained Multilingual Representations (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have focused on cross-linguality of contextual embeddings . however, they are only moderately language-neutral by default . |
| Approach: | They propose to use unsupervised centering and fitting an explicit projection on parallel data to achieve stronger language neutrality. |
| Outcome: | The proposed model outperforms existing models on XNLI and NER tasks. |
LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation (2020.lrec-1)
Copied to clipboard
| Challenge: | Recent trends in NLP research have raised an interest in linguistic code-switching . however, many of these approaches are limited to a few language pairs and a specific domain . |
| Approach: | They propose a centralized benchmark for Linguistic Code-switching Evaluation that combines eleven corpora covering four different code-switch languages and four tasks. |
| Outcome: | The proposed benchmark provides a centralized benchmark and compares with other benchmarks in real-time. |
A Recipe of Parallel Corpora Exploitation for Multilingual Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have highlighted the potential of exploiting parallel corpora to enhance multilingual large language models. |
| Approach: | They investigate the impact of parallel corpora quality and quantity, training objectives, and model size on performance of multilingual large language models enhanced with parallel corporeal. |
| Outcome: | The proposed approach improves performance in bilingual and general-purpose tasks. |
Improving Language Identification for Code-Switched Speech: The Pivotal Role of Accented English (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing models fail to identify English spoken with the accent of the matrix (dominant) language. |
| Approach: | They propose to fine tune existing LID models with accented English to improve code-switched LID . they use a metric that captures relative ranking of identified languages often overlooked by traditional metrics. |
| Outcome: | The proposed model can be fine tuned with small amounts of accented English without degrading performance on monolingual speech. |
Fumbling in Babel: An Investigation into ChatGPT’s Language Identification Ability (2024.findings-naacl)
Copied to clipboard
| Challenge: | ChatGPT is a powerful NLP tool but its language identification abilities are unclear. |
| Approach: | They compile a benchmark comprising 670 languages representing 23 language families spoken in five continents and compare their language identification abilities to ChatGPT's (both GPT-3.5 and GPT-4) performance. |
| Outcome: | The proposed model performs poorly on African languages, while GPT-3.5 and GPT-4 perform poorly on English, Afrikaans, Arabic, Indonesian, Italian, Mandarin Chinese, and several more. |
Where are you from? Geolocating Speech and Applications to Language Identification (2024.naacl-long)
Copied to clipboard
Patrick Foley, Matthew Wiesner, Bismarck Odoom, Leibny Paola Garcia Perera, Kenton Murray, Philipp Koehn
| Challenge: | Language identification (LID) is a critical component in many modern multilingual speech technologies. |
| Approach: | They propose to use radio broadcasts with known origin to train regression models . they also propose to explore using geolocation as a proxy task for LID . |
| Outcome: | The proposed model outperforms pretrained models on the FLEURS benchmark and on the VoxLingua benchmark. |
Unsupervised Preference-Aware Language Identification (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies do not consider inter-personal variations due to the lack of user annotated training data. |
| Approach: | They propose to use user preferences to identify ambiguous texts in multilingual applications without user annotated training data to build a preference-aware LID model. |
| Outcome: | The proposed model significantly outperforms existing LID systems on handling ambiguous texts. |
An Exploratory Analysis of the Relation between Offensive Language and Mental Health (2021.findings-acl)
Copied to clipboard
| Challenge: | Using computational models, the use of offensive language is pervasive in social media . a popular line of research is the study of machine learning classifiers to identify offensive content online . |
| Approach: | They analyze social media posts written by individuals with depression and those without . they train computational models to compare use of offensive language with depression detection . |
| Outcome: | The proposed models show that offensive language is more frequently used in the samples written by individuals with depression and those showing signs of depression. |
MHE: Code-Mixed Corpora for Similar Language Identification (2022.lrec-1)
Copied to clipboard
| Challenge: | a new corpus of code-mixed data-sets is presented for similar language identification . the data-settings are based on a more-resourced minority language, Magahi . |
| Approach: | They propose a Magahi-Hindi-English code-mixed corpus for similar language identification . they discuss the complexity of the data-set and provide a few baselines . |
| Outcome: | The proposed corpus provides a language id at two levels: word and sentence. |
CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages (2024.lrec-main)
Copied to clipboard
Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, Thien Huu Nguyen
| Challenge: | Existing training datasets for large language models are often not fully disclosed. |
| Approach: | They propose a multilingual dataset with 6.3 trillion tokens in 167 languages . they use a pipeline of multiple stages to achieve the best quality for model training . |
| Outcome: | The proposed dataset is cleaned and deduplicated to achieve the best quality for model training . lack of transparency has hindered research on attributing and addressing hallucination and bias issues . 6.3 trillion tokens in 167 languages are used to train multilingual LLMs . |
Typological Features for Multilingual Delexicalised Dependency Parsing (N19-1)
Copied to clipboard
| Challenge: | Existing universal models to describe the syntax of languages are debated for decades . a new study examines the plausibility of universal grammars in dependency parsing . |
| Approach: | They propose to use typological features to describe the syntax of languages to train a multilingual dependency parser. |
| Outcome: | The proposed model can be trained on 40 languages with the help of typological features. |
GlotLID: Language Identification for Low-Resource Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing web-mined datasets for low-resource languages have been useful for low resource NLP. |
| Approach: | They propose a model that identifies 1665 low-resource languages and a new model that is rigorously evaluated and reliable. |
| Outcome: | The proposed model outperforms baselines when balancing F1 and false positive rate (FPR). |
HeLI-OTS, Off-the-shelf Language Identifier for Text (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing off-the-shelf language identification tools favor widely used languages, but Heli-OTS can be used to identify a large group of languages. |
| Approach: | They introduce an off-the-shelf text language identification tool using the HeLI method . they compare the He LI-OTS language identifier with fastText on two different data sets . |
| Outcome: | The proposed language identification tool is compared with fastText on two different data sets. |
On The Performance of Time-Pooling Strategies for End-to-End Spoken Language Identification (2020.lrec-1)
Copied to clipboard
| Challenge: | Language identification (LID) from speech is commonly tackled using similar approaches to those employed for speaker verification/recognition. |
| Approach: | They propose to combine local descriptors and global descriptores into a single global description that can be used for downstream classification tasks. |
| Outcome: | The proposed methods outperform well-known benchmark systems and previously results based on attention only. |
OpusTools and Parallel Corpus Diagnostics (2020.lrec-1)
Copied to clipboard
| Challenge: | Currently OPUS contains 57 released corpora covering over 700 languages and language variants creating more than 70,000 bitexts in the sense of aligned language pairs across all corporata. |
| Approach: | They introduce OpusTools, a package for downloading and processing parallel corpora in OPUS . the package implements tools for accessing compressed data in their archived release format . they show how they can be used in parallel corpus creation and data diagnostics . |
| Outcome: | The proposed tools can be used in parallel corpus creation and data diagnostics. |
Common Voice: A Massively-Multilingual Speech Corpus (2020.lrec-1)
Copied to clipboard
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, Gregor Weber
| Challenge: | Common Voice is a massively-multilingual collection of transcribed speech intended for speech technology research and development. |
| Approach: | They propose to use Mozilla’s DeepSpeech Speech-to-Text toolkit to perform multilingual automatic speech recognition experiments. |
| Outcome: | The proposed corpus is the largest in the public domain for speech recognition, both in terms of hours and languages. |
Huqariq: A Multilingual Speech Corpus of Native Languages of Peru forSpeech Recognition (2022.lrec-1)
Copied to clipboard
| Challenge: | the Huqariq corpus is a multilingual collection of speech from native Peruvian languages . the project is designed to preserve endangered languages in the public domain . |
| Approach: | They propose to use crowdsourcing to collect transcribed audio from native Peruvian languages . they propose to do 220 hours of speech recognition experiments to verify quality . |
| Outcome: | The Huqariq corpus is a multilingual collection of speech from native Peruvian languages . the project is expected to reach 20 native languages out of 48 native languages by 2022 . |
OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification (2024.acl-long)
Copied to clipboard
| Challenge: | Autoregressive models can be slower during inference and have potential risks of hallucination. |
| Approach: | They propose an encoder-only speech foundation model based on Connectionist Temporal Classification. |
| Outcome: | The proposed model improves on 180k hours of public audio data for multilingual speech recognition, speech translation, and language identification. |
Geographically-Informed Language Identification (2024.lrec-main)
Copied to clipboard
| Challenge: | a paper develops a method to identify languages based on geographic origin of text . the model is based in regions where languages are widely spoken and may occur anywhere . |
| Approach: | They propose to incorporate geographic information into a language identification model to ensure coverage of linguae francae regardless of location. |
| Outcome: | The proposed model includes 31 widely-spoken international languages . the model improves on social media data and improves performance on 916 languages compared to baseline models . |
VAST: A Corpus of Video Annotation for Speech Technologies (L18-1)
Copied to clipboard
| Challenge: | The video annotation for speech technologies corpus contains 2900 hours of video data . the data are intended to support speech technology development . |
| Approach: | The Video Annotation for Speech Technologies corpus contains 2900 hours of video data . the data are intended to support speech technology development . |
| Outcome: | The video annotation for speech technologies corpus contains 2900 hours of video data . the data are intended to support speech detection, language identification, speaker identification, and speech recognition . |
Language Identification for Austronesian Languages (2022.lrec-1)
Copied to clipboard
| Challenge: | This paper provides language identification models for low- and under-resourced languages in the Pacific region with a focus on previously unavailable Austronesian languages. |
| Approach: | They compare a classifier based on skip-gram embeddings with other methods . they then increase the number of non-Austronesian languages to 800 to evaluate their performance . |
| Outcome: | The proposed model improves on the previous methods for low- and under-resourced languages in the Pacific region. |
GeezSwitch: Language Identification in Typologically Related Low-resourced East African Languages (2022.lrec-1)
Copied to clipboard
| Challenge: | Low-resourced languages with similar typologies are often confused with each other in real-world applications such as machine translation, affecting the user’s experience. |
| Approach: | They propose to build a dataset for five typologically and phylogenetically related low-resourced East African languages using the Ge’ez script as a writing system. |
| Outcome: | The proposed dataset is built automatically from selected data sources, but also performed a manual evaluation to assess its quality. |
From English to Code-Switching: Transfer Learning with Strong Morphological Clues (2020.acl-main)
Copied to clipboard
| Challenge: | Linguistic code-switching (CS) is an understudied area in natural language processing . lack of resources and annotated data makes it difficult to strive for progress in CS-related tasks. |
| Approach: | They propose a method to adapt monolingual models to code-switched text in various tasks . they transfer English knowledge from a pre-trained ELMo model to different code-paired languages . |
| Outcome: | The proposed method outperforms multilingual BERT and homologous CS-unaware models and provides state-of-the-art in CS tasks. |
Humanitarian Corpora for English, French and Spanish (2024.lrec-main)
Copied to clipboard
| Challenge: | et al., a leading database of humanitarian documents, compiled with ReliefWeb reports . documents selected with language identification and noise reduction techniques . authors present corpora of English, French and Spanish humanitarian documents . |
| Approach: | They present three corpora of English, French and Spanish humanitarian documents compiled with ReliefWeb reports . documents were tokenized, lemmatized, tagged by part of speech, and enriched with metadata . authors propose a project to develop a humanitarian dictionary with a focus on conceptual variation . |
| Outcome: | The corpora were compiled to satisfy the research needs of the Humanitarian Encyclopedia project with a focus on conceptual variation. |
Script Normalization for Unconventional Writing of Under-Resourced Languages in Bilingual Communities (2023.acl-long)
Copied to clipboard
| Challenge: | linguistically under-represented communities have an extraordinary opportunity to create content in their native languages. |
| Approach: | They propose to solve the problem of script normalization for languages written in a Perso-Arabic script and use a transformer-based model to analyze the noise levels. |
| Outcome: | The proposed model can normalize a language written in a Perso-Arabic script and improve machine translation and language identification tasks. |
Language and Speech Technology for Central Kurdish Varieties (2024.lrec-main)
Copied to clipboard
| Challenge: | a recent study focused on the Kurdish language, a less-resourced Indo-European language spoken by over 30 million speakers. |
| Approach: | They propose to develop resources for language and speech technology for Kurdish . they report the performance of machine translation, automatic speech recognition and language identification . |
| Outcome: | The proposed model is based on transcribing movies and TV series as an alternative to fieldwork. |
Identifying Open Challenges in Language Identification (2025.acl-long)
Copied to clipboard
| Challenge: | Existing work on language identification has focused on cross-domain setups, but no systematic comparison is available. |
| Approach: | They propose to train an accurate multi-domain languageidentification model on 2,034 languages and analyze the remaining errors. |
| Outcome: | The proposed model performs well on 2,034 languages with training with 1,000 instances per language and a maximum input length of 100 characters. |
LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Currently, existing systems cannot accurately identify most of the world's 7000 languages due to lack of data and computational challenges. |
| Approach: | They propose a misprediction-resolution hierarchical model, LIMIT, that reduces error by 55% on a children's stories dataset and by 40% on 'fLORES-200' benchmark. |
| Outcome: | The proposed model reduces error by 55% on the MCS-350 and 40% on the FLORES-200 benchmarks. |
NusaAksara: A Multimodal and Multilingual Benchmark for Preserving Indonesian Indigenous Scripts (2025.acl-long)
Copied to clipboard
Muhammad Farid Adilazuarda, Musa Izzanardi Wijanarko, Lucky Susanto, Khumaisa Nur’aini, Derry Tanti Wijaya, Alham Fikri Aji
| Challenge: | NusaAksara covers 8 scripts across 7 languages, including low-resource languages not commonly seen in NLP benchmarks. |
| Approach: | They propose a benchmark for Indonesian scripts that includes their original scripts and a dataset that includes 8 scripts across 7 languages. |
| Outcome: | The proposed benchmark covers 8 scripts across 7 languages, including low-resource languages not commonly seen in NLP benchmarks. |